The Concept of a Visual Interface for Conducting Humans and Synthesizers

A novel approach for visual control of electronic digital music instruments is proposed, which is suitable for being applied in mixed ensembles of human musicians and electronic instruments. Gesture tracking and baton tracking by established computer vision methods provide input of either direct musical parameters (volume, pitch, expression) for the sound creation (control of synthesizer instruments) or serve as cues for the time-synchronized replay of previously recorded music sequences (beat, tempo, expression). In conjunction with acoustic signal processing, such a system could enable the automatic playing of a computer-based member of an orchestra, in which the conductor conducts both the instrument as well as the human musicians. This approach allows an intuitive control of the timing and expression of stored digital music, allowing the creation of a unique interpretation. This paper lays out the basic idea and concept.